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The optimum linear modulator and demodulator which -provide transmis- 
sion of a gaussian vector source through an additive gaussian vector channel 
are derived in this paper. The measure of performance that is used is 
the transmission distortion, which is defined here as the mean square error 
between the source output and the decoder output. It is assumed that the 
source and channel are mutually independent but that correlations can 
exist among the components of each. The performance of the best linear 
system is then compared with the distortion shown by Shannon to be 
theoretically obtainable when no functional constraint is imposed at the 
modulator other than an energy constraint. Although the precise form of 
this optimum modulator is not known for general gaussian vector sources 
and channels, it is known to be nonlinear and to require arbitrarily long 
coding block lengths. However, it is a commonly held notion that when 
the source and channel dimensionalities are equal the optimum modulator 
is linear and requires a block length of only one. It is shown here that 
this belief is incorrect except in very particular situations which are de- 
scribed. Some relations between the optimum linear modidator-demodulator 
pair and Shannon's test channel are discussed, and an example is in- 
cluded which shows that the nonoplimality of linear devices can be quite 
small. 

I. INTRODUCTION 

We are concerned here with the transmission of a gaussian vector 
source over an additive gaussian vector channel. The mean square 
difference between the source and decoder outputs is used to measure 
the transmission distortion in the system and is, therefore, attempted 
to be minimized in the design of the encoder and decoder. In this 
design the encoder is constrained to present only a limited energy to 
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the channel, thus constraining the transmission capacity of the system. ' 
It is because the transmission capacity of the system is limited in this 
way that the given gaussian vector source cannot be transmitted with 
arbitrarily small error. 

The distortion which necessarily must exist in the system is pre- 
scribed by Shannon's rate-distortion theory. 2 This theory states that 
when the transmission rate in a system is limited to R, the transmission 
of the source must include an average distortion of at least d R , which 
in general is a function of the source statistics and the distortion measure. 
The theory further states that the distortion level d R is attainable 
with some modulator-demodulator pair. Unfortunately, the precise form 
of this modulator and demodulator is not known in general, except 
that it is nonlinear 3,4 and that it requires the use of arbitrarily long 
coding block lengths. 2 

Since the nonlinearity of the optimum encoder is probably a very 
complex twisting of the source space locus within the channel input 
space, the implementation of the optimum encoder, even if it were 
known, would be extraordinarily complex. Of course, the long coding 
block length requirement does nothing to help the situation. For these 
reasons we study in this paper the optimum linear transmission system, 
restricting both the encoder and decoder to be linear operators. Such 
a system uses a block length of only one and is very simple to implement. 
(It is later shown that increasing the block length does not improve 
the performance.) 

The degradation in performance with the use of the optimum linear 
system is found by comparing the resulting distortion to that of the 
optimum nonlinear system as found by Shannon. Contrary to popular 
belief, the best linear system does not provide the minimum attainable 
distortion, even when source and channel dimensionalities are equal, 
except in very particular situations that are described. However, in 
many cases the difference is small. At the end of the paper we discuss 
some relations between the optimum linear modulator-demodulator 
pair and Shannon's test channel. 2 

II. THE LINEAR TRANSMISSION SYSTEM 

The system considered is shown in Fig. 1. The N, dimensional 
zero-mean source vector w is linearly modulated by A to form the 
input to the N e dimensional additive gaussian noise channel. We 
assume the noise vector n to be independent of w. The linear demodu- 
lator B extracts from the received vector y an estimate w of the source 
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Fig. 1 — The linear system. 

which is presented to the user. In summary 

v) = By = B(x + n) = B(Aw + n). 



(1) 



The measure of distortion in the system is taken to be the sum 
of mean-square errors between the components of iv and w, that is 



d = E[\w - w | 2 ] = e\ £ {w t - wtfY 



(2) 



The modulation matrices, A and B, are sought which minimize this 
distortion, their choice subject only to an average channel input energy 
constraint, 



St = E\ X> 2 = SVar^ , 



(3) 
(4) 



^ iSo, 

which obviously will be met with equality in the optimum system. 

It is well known that the minimum mean square error estimate of 
any quantity (here the source vector iv) based on the observation 
of a second quantity (here the channel output vector y) is the condi- 
tional expected value of the first given the second. 4 Further, the average 
error made with such an estimate is the conditional variance of the 
first given the second. Therefore, we have 

u>,- = E(Wi | y)\ i = 1, 2, ••• , N, 



(5) 



d = ^2 Var (w { \ y). 
The required conditional density p(w\y) can be found from 
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p(w) = fci exp [— iw'ft'Hit] 

and 

p(y \w) = k» exp [-£(# - AwY$; l (y - Aw)] 

by application of Bayes rule. The result is 

p(w | y) = fc 3 exp \-\{w - w)'^) v (w - iu)] 
with 

$-;„ = a 1 *; 1 a + <*>;' (6) 

and 

w' = y'Q^A^y . (7) 

From these equations we have one immediate result, that is, the optimum 
demodulator matrix is given in terms of A by 

B = •.„A , «T l . (8) 

If we now rewrite equations (5) and (3) as 

d = trace <£>„,,„ (9) 

S T = trace $ x (10) 

we can restate our problem as that of finding the matrix A which 
minimizes the trace of $„!„ subject to a constrained maximum trace 
of <*>* . 

III. THE SOLUTION UNDER CERTAIN ASSUMPTIONS 

We first restrict our attention to systems in which the source and 
channel dimensionalities are equal, N. = N e = N, and in which the 
correlation matrices $„ and $„ are diagonal. From equation (6) we 
have 

*«*;!,, = Qj^VfA + I (11) 

and from equation (1) that $, = A$ W A' and <f>„ = * x + $„ , which 
provides 

$ B fri = A*„A'*; 1 + /. (12) 

Noting that $ y enters these equations in a more symmetric way than 
dose *„ , we recast the energy constraint in equation (10) to be in 
terms of the received energy at the channel output. This energy equals 
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s --*[ijtf] = % Yariji 



= trace 4>„ 

= trace $ a + trace f>„ 
which, if trace $„ = N , is constrained to satisfy 

S R ^ So + No . (13) 

3.1 The Proof that the Optimum Modulator Matrix is Diagonal 

If we denote the characteristic polynomial of a matrix M in the 
variable X by 

c.p. [M, X] = det (M - XI) 

and state that M { is square, we can use the following two matrix 
properties: 5 

(i) c.p. [M,M 2 , X] = c.p. [il/ 2 M, , X] (14) 

(ii) c.p. [Mt , X] = c.p. [M t + /, X - 1] (15) 

to conclude from equations (11) and (12) that 

c.p. [#„#;!„ , X] = c.p. [#„$;•, X]. (16) 

It is this equation which provides the important relations among the 
correlation matrices in the system. 

We note that the set of matrix pairs ^ wiy , <£„ which are consistent 
with equation (16) include man}' pairs which do not satisfy both equa- 
tions (11) and (12) for any given A. The latter equations of course 
specif}' the relations among #„(,, and <i>„ which must exist in the com- 
munication problem under consideration. Nevertheless, we will work 
with equation (16) to perform the optimization and then show that 
the solutions for $„\ y and $ y can be realized with some modulator 
matrix A and, therefore, are consistent with the more restrictive equa- 
tions (11) and (12). 

Equation (14) and the assumed diagonal form of <J>„. and <f>„ allows 
us to rewrite equation (16) as 

c.p. [$l*;! y $i , x] = c.p. [*;**„*;*, \]. 

As <& w and 4>„ are system constants not under the control of the user, 
any specification of $„ completely determines the roots of $£*$„<!£*, 
which we denote by {a,-}, i = 1, 2, • • • , N. The roots of $;*$„|„$;* 
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are also determined and are equal to fa; 1 }. Our claim now is that 
among all matrices $ with roots {o^ 1 }, the one which produces the 
minimum trace of $„,,„ = $j$$j is diagonal. 

If tpn are used to denote the elements of $, the trace of $ wly equals 

N 

trace $ w \ v = ^ o&u . 

i-l 

At this point we impose, without loss of generality, that the variances 



be ordered such that <t\ ^ a\ ^ 



^ 9m 



Since the minimum 



trace of *„,„ is sought, clearly the diagonal elements tp ti should cor- 
respondingly satisfy <pn ^ <p 22 ^ • • • ^ <Pnn • This presents no re- 
striction on $ as a simultaneous interchange of rows and columns 
produces no change in the characteristic equation of $. 

Now consider any nondiagonal candidate for the desired <£. In 
particular, let <p mk = (p km , m > k, be nonzero. Because the submatrix 



$(fcm) = 



<Pkk 'Pkm 

L<Pmk VmmJ 

is itself a correlation matrix, it can be diagonalized by some orthogonal 



matrix T such that 

*'(/fcm) = T$(km)T = 



<Pkk 








(Cmm. 



From (14) it is known that the characteristic polynomials of $(fcm) 
and &(km) are equal. The trace and determinant of each are therefore 
equal. It follows that <p' kk = f kk — c and <p' mm = <p mm + c; c > 0, or that 
the larger diagonal element is increased and that the smaller one is 

decreased. 

The diagonalization of the submatrix $(/cra) within $ can be effected 
by an orthogonal matrix Q which contains T in the appropriate sub- 
matrix position and identity matrix elements in the other positions: 

qa = Ui ; (*i i) = (&. k), (k, m), (m, k), (m, m) 

q ti = S if ; other (i,j). 

We then have <£>' = Q$Q* with only the elements in $' in rows and 
columns k and m changed from those in $. If $' is used to generate 
a new correlation matrix *i lf = $j$'<i>J , we have 



tr *;„ = £ ab/ tl = Z 



(Ti<Pi 



/ 2 2 \ 

— c{a k — <j m ) 
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= tr *,, | - C (al — al) 

£tr *,,.,„, (17) 

which establishes the claim of this section. That is, any nondiagonal 
correlation matrix <£ with roots {aj 1 } conjectured as providing a 
minimum trace correlation matrix ^j^^j = $„|, can be improved 
upon by $>'. The desired matrix for <i> is therefore diagonal and equal to 

* = [oT 1 !,,] (18) 

with the corresponding form of <t>„,| y equal to 

#.„ = [^-'5,,]. (19) 

It follows that among all matrices $ w \ u consistent with equation (16) 
with any given <£„ , the one with minimum trace is diagonal. 

An identical argument yields the symmetric conclusion. That is, for 
any specified $„,,„ the matrix $„ with minimum trace among those 
consistent with equation (16) is also diagonal and equal to 

<f>„ = [<&*,«„]. (20) 

The argument assumes only that the noise variances are ordered 
oj, £ a' ni ^ • • • g ff' n/t . 

We can now state that the minimization of the trace of $„!„ over 
all pairs $„(„ , $„ which satisfy equation (16) and the constraint equa- 
tion (13) is obtained with a pair of diagonal matrices parametrically 
related as in equations (19) and (20). Any pair not so related can be 
altered, one matrix at a time, to decrease either the error (trace $ w \ v ) 
or the received energy (trace <!>„). Although we have worked with 
pairs <!>„,|„ , <!>„ consistent with equation (16) rather than the smaller 
set satisfying equations (11) and (12), the solution forms for ^i,, and 
$„ are still valid as they do satisfy these equations. 

The modulator matrix which produces the correlation matrices $„\ u 
and <£„ in the optimum form can be found from either equation (11) 
or (12) to be 



-fefc-D 1 *} 



a = |_^r(«. - 1)5 *'J" (21) 

Equations (12), (14), and (15) and the fact that *~*A*»4. I *~* has 
nonnegative roots (it is a correlation matrix) can be used to show 
that <x t ^ 1, i = I, 2, • - • , N which guarantees that the elements 
of A are real. It remains to solve for the set of roots (a,-) which provides 
the desired optimization. 
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3.2 The Optimum Diagonal Modulator Matrix 

In terms of the set {a,}, the distortion which is to be minimized 
is given by 

N 

d = trace $ wly = ^2 ofe 1 

i-l 

and the received energy constraint by 

N 

S R = trace $„ = 23 °""' a «" ^ S + N n . 

A further constraint is that a t ^ 1, i = 1, 2, • • • , N. As the set of 
permissible a.'s is a convex set and the functions d(a t ) and S R (a t ) are 
convex functions, the Kuhn-Tucker theorem is applicable. 8 This states 
that at the point of minimization: 



dot; 



;[<* + £«*] 



= if a, > 1 
< if a, = 1. 



Therefore we have 



-<r?a7 2 + A*li = if a, > 1 
A 

< if a,- = 1 



or 



«,- = max 



cr,- 
LVXcr ni 



. ] 



(22) 



It has already been observed that a^ ^ a 2 ^ • • ■ ^ «w and that 
a,- = 1 corresponds to a,< = or no transmission of the ith source 
component. If we let N' denote the last a, strictly greater than one 
we have the following solution for the optimum modulator matrix 



A = 



m- 1 ' 5 " ° 



L 











1 ^ i, j ^ N'. 



(23) 



The solution for the distortion in the optimum linear system follows 
directly from equation (19): 



AT' N 

d = 2 ^ViV-i + Z3 ff2 > i 

i-l i-Af* + 1 



(24) 
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as does the solution for the total received energy from equation (20) : 

A • (25) 



AT < ff 

i~\ A 



N' + \ 



In these equations, the parameter X is chosen to satisfy the constraint 
in equation (13) with equality. It should be remembered in the solu- 
tion for X that N' is a function of X, being equal to the largest value 
of i for which <r { /a mt = a. For completeness, we give the optimum de- 
modulator matrix: 



B = 



&i 



^- 1]s " ° 



1 ^ i, j £ N'. 



(26) 



IV. ELIMINATION OP THE ASSUMPTIONS 

4.1 A Source and Channel with Nonindependent Components 

We now consider systems in which $,„ and <i> n are not diagonal. 
Let P and R be the orthogonal matrices which respectively diagonalize 
these two correlation matrices, that is, <fv = P$ w P l and #„- = R$ n R' 
with <£„,< and $„. diagonal. Using the previous results, we can find the 
optimum modulator matrix A' in the primed system containing the 
correlation matrices <£,„. and $„< • Now consider the use of the modulator 
matrix A = R l A 'P in the system with $„, and <£„ . From equation (6) 
and $>„ = A$ W A' + <p„ , it can be easily shown that using A' in the 
primed system and A in the unprimed system each produces the same 
distortion and uses the same energy. Consequently, A must be the 
optimum matrix in the unprimed system. If it is not, and A Q is better, 
A' = RA Q P l would be a better choice than A' for modulator in the 
primed system contrary to A' being optimum. 

4.2 Nonequal Source and Channel Dimensionality 

When N, ^ N c , we can appropriately modify either the source or 
channel to restore the equality. For example, when N, < N e , N e — N. 
source components of arbitrarily small variance, say e, are added to 
the original source vector. The optimum modulator is then found as 
a function of e by the previous method, and finally the limit taken 
as e goes to zero. Similarly, when iV c < N, , N, — N c , channel com- 
ponents of arbitrarily large noise variance, say 1/e, are added to the 
original channel, the optimum modulator found, and the limit taken 
as e goes to zero. We have seen that whenever either the source has 
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a component with small variance or the channel has a component 
with large noise variance, the number of source components actually 
transmitted, N', is smaller than N. Since the optimum modulator 
matrix is diagonal, N' is also the number of channel components 
actually used. Therefore, the limiting modulator form in both of the 
above situations is attained for a nonzero value of e, say ei . This 
modulator form is then optimum for all e < t t 7^ 0. 

V. COMPARISON OF OPTIMUM LINEAR AND NONLINEAR MODULATORS 

In 1959 C. E. Shannon introduced a relation between d R , the min- 
imum attainable transmission distortion of a source, and R, the total 
information rate used in transmission. 2 This relation involves only the 
source statistics and the distortion measure in use. From it one is 
able to conclude that any channel with capacity R can be used to 
transmit the source with a transmission distortion arbitrarily close 
to d R . One need only use a "sufficiently complex" encoder and decoder. 

Another part of rate-distortion theory is the idea of a "test channel." 
Associated with each point on the rate-distortion curve, (d R , R), is 
such a test channel which has the significance that among all channels 
that transmit the source at a rate equal to R, it provides the minimum 
transmission distortion d R . Therefore, if there exist pre- and post- 
operators which can transform a given capacity R channel into the 
test channel for the source at (d R , R), these operators must be optimum. 
An obvious necessary condition for this transformation, which is not 
always met, is that the capacity of the test channel at (d R , R) be equal 
toR. 

For a gaussian source with variance a 2 and squared difference distor- 
tion, Shannon has found 2 both the rate distortion expression, d R = a 2 e~ 2R 
and the test channel: 

w <- <- ■&. (27) 

T 



In this reverse channel, w and n are independent gauss variables with 
respective variances a 2 — d R and d R . It can be shown that this channel 
is identical to the forward channel: 

w -> (x) -» © -* w (28) 

T T 



A, 
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with Ai = (a 2 — d R )/a 2 , o- 2 = A x d R , and the independence between 
w and n. A similar form is given by Gallager in Ref. 7. Still another 
form of the test channel is: 

w->®->®->(g)->M> (29) 

T T T 

A n B 

with A 2 = (o- 2 - d R )o 2 J<j 2 d R , B 2 = (<r 2 — d R )d R /o 2 o 2 n , and n any 
given additive gaussian noise. 

Now consider a single dimensional gaussian channel of capacity R. 
Since the received energy S R is accordingly restricted to a\ exp (2R), 
we have from equations (23) through (26) that the optimum linear 
operators are 

2 a 2 , ( o" -\ cr'„ ( a — d 

fl " = 7 te " V = 7 V~d~ 
ft = x'(f- - i) = ^V^ 

\A(r n / (t a n 

d aa n 

A — — „ • 

<T(T n O/j 

Note that the distortion d equals a 2 exp (2R), and that a n and 6 n 
agree precisely with the test channel parameters in (29). Therefore, 
we can conclude that in this case the operators in equations (23) and 
(26) are optimum, even outside the linear class. 

The rate-distortion curve and the test channel for gaussian vector 
sources can also be found from Shannon's results. The results for the 
iV-dimensional source with variances a\ , a\ , ■ ■ • , al are (we continue 
to assume that a\ ^ a\ ^ • • • ^ a, 2 ) : 

JV "\ 1 AV 



d R = N\e- 2R II «r?j ; g d R ^ N*l 

(N-l \ 1AV-1 

e~ 2R II *'ij ; 



Nal ^d R ^ 4 + (N - 1)4_, 



Ar-2 "| 1AV-2 

= 4 + 4-, + (tf - 'Me'** II *5 



ff.v 



, + (N - 1)4-. ^ rf* ^ 4 + 4-i + (N - 2)4-2 
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2 , i 2 . 2 -2fi. 

= a N -t • • • + <r 2 + o"i« ; 

ON + • • * + 2(T2 ^ d* ^ o& + • • • + ff? 

- a% + • • • + *\ ; R = 0. (30) 

This expression can also be applied to a gaussian vector source with 
correlated components if the variances a\ are interpreted as those in 
the diagonalized correlation matrix «£„,. = P$ W P'. The test channel 
for N > 1 is the product of elementary test channels given in (29) with 

A = A.\ , A.2 , ■ • ■ , Ajv , 

42 _ (gj — d { ) 2 

Ai — 2 i Oni 1 

d t = min (cri , d R ), 

al = ali , al 2 , ' ' • , <Kn — an Y noise vector. 

Let us now presume that the vector channel provided for use has 
the additive noise variances given by the vector a 2 „ and is constrained 
to have an output energy level equal to S R . This equivalently specifies 
the channel capacity as 

R = max £ I log % (31) 

with 

iS a< = max (*S, alt) 

and <S adjusted to have ^2 S Ri = S R . The comparison between the 
minimum attainable transmission distortion using linear transmitter 
and receiver operators (equation 24) and using unrestricted transmitter 
and receiver operators (equation 30) now reveals that contrary to the 
single dimension case, when N > 1 the linear operators are not, in 
general, optimum. The only exception is when both the vectors a 2 
and al are uniform. Some intuition as to why the single and multi- 
dimensional cases are different might be provided by the following. 

The test channel at (d R , R), for example, the one including the 
noise vector a\ in its form, is a result of a minimization of mutual 
information under a distortion constraint. It does not, therefore, 
necessarily divide the total energy presented to the gaussian vector 
channel in a way which uses this channel to capacity. Since this channel, 
by definition, transmits information at a rate equal to R, its total 
capacity is (except for the special case noted previously) strictly 
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greater than R. Consequently, when the same additive noise channel, 
<t\ , is to be used for transmission but is stated to have a capacity 
of only R, it cannot be transformed into the test channel by any pre- 
and postoperators. 

The impossiblity of such a transformation can also be observed 
by noting that the allowed total input energy on the given capacity 
R channel is restricted to a lower level than present on the test channel 
The uniqueness of the test channel, which is formed with linear op- 
erators, and the continuity of both the mutual information and dis- 
tortion with the modulator matrix then precludes the possibility of 
attaining the test channel's performance with the given capacity R 
channel and linear operators. 

One could argue that the comparison to this point is not fair in 
that Shannon allows modulators and demodulators that operate on 
blocks of letters, whereas the results in equations (23), (24), and (25) 
were derived using a coding block length of one. However, the previous 
results show that the optimum linear modulator does not mix indepen- 
dent source components before presentation to the channel, assuming 
the channel has already been rotated in iV-space so as to have indepen- 
dent noise components. Neither does it cross-couple sets of source 
components having no cross dependence when presentation is to a 
channel with sets of noise components of equal respective dimen- 
sionalities also having no cross dependence. Therefore, if successive 
source and channel (vector) events are independent, and their dimen- 
sionalities filled out to be equal by adding either zero variance source 
components or infinite variance noise components, there is no memory 
introduced by the optimum linear modulator among elements of the 
encoded block. The consequence is that the distortion and the energy 
are only scaled by the block length in use. 

VI. AN EXAMPLE 

We cite here just one example which shows that at least in many 
cases the performance of the optimum linear modulator- demodulator 
pair compares favorably with that theoretically obtainable with more 
complex operators. We take tr t = tr 2 = 1, a nl = a, a n2 = ae 2 * and use 
a and <p as parameters that generate a set of different channels. To 
better compare the two performances, we fix the channel capacity at 
C which in turn fixes Shannon's minimum attainable distortion at 
d c = 2e~ c . The total allowed received energy is thus specified ac- 
cording to equation (31). 

Upon solution for X and d in equations (24) and (25) we have the 
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following expression for the ratio between the distortion obtainable 
with linear operators and that theoretically attainable: 



d c 



cosh 2 <f 
cosh 2 <p 



cosh (2<p - C) ' 



g <p ^ \C 
hC S<p ^ C 



cosh C 



C < 



^ <P- 



We illustrate this function for several different values of capacity in 
Fig. 2. At <p = (where both the vectors a~ and <r 2 are uniform) we 
see that d(0) = d c indicating the optimality of the linear modulator 
and demodulator for this case. Using a term introduced in Ref. 8, 
we can therefore say that when <p = the source and channel are 
"matched." As <p increases, the source-channel mismatch increases and 
the nonoptimality of linear operators also increases. As the figure 
illustrates, the nonoptimality ratio, d{tp)/d c , can be quite large when 
both the channel capacity is high and the additive noise vector is 
highly skewed in variance. However, over a significant region of interest, 




Fig. 2 — The linear Bystem nonoptimality for iV = 2, a x = a 2 = l,e»i = l,«r„j = exp 2</>. 
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<p ^ 1 (reflecting a noise component variance ratio of about 50), the 
nonoptimality ratio is small. 

VII. SUMMARY 

In this paper we have derived the optimum linear modulator and 
demodulator for the transmission of a gaussian vector source through 
an additive gaussian vector channel. It was found that when both 
the source and channel components are independent, both the modulator 
and demodulator matrices are diagonal. This specifies the separate 
amplification, transmission, and decoding of each source component. 
When both the source and channel components are correlated, the 
optimum modulator matrix was found to be the cascade of three 
matrices: (i) the orthogonal matrix which diagonalizes the source 
correlation matrix, (u) the optimum modulator matrix which transmits 
this newly formed independent component source over the independent 
component additive noise channel which is formed by {Hi) the orthog- 
onal transformation matrix that diagonalizes the noise correlation 
matrix. We have found that in general the best linear system does not 
provide a distortion as small as that stated by Shannon to be attainable 
with a channel of the same capacity. The only exception is when both 
the source and channel noise variance vectors are uniform. The non- 
optimality of linear modulators and demodulators can be quite large 
in some cases but, in many other situations, can be small enough to 
justify the use of these very simple operators. 
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